Method and system for context dependent pop-up menus

ABSTRACT

A method and system for providing context dependent pop-up menu icons that activate menu icons in response to a select &amp; drag operation by manipulation of a pointing device. Presented icons may correspond to functions or actions that can be implemented with the particular type of content selected. Icons may be displayed in a consistent manner in terms of position and content with respect to the selected file or object. Icons may appear only during a select &amp; drag operation, thereby leaving the display uncluttered at other times. Presentation of icons may be discontinued when the selected file or object is dragged beyond the area of the presented icons. New applications can register with the operating system to specify icons to be presented when corresponding files are selected or when particular content objects are selected within the application.

FIELD

The present invention relates generally to computer user interfacesystems and more particularly to methods and systems for providingcontext-dependent drag and drop menus within a graphical user interface.

BRIEF DESCRIPTION OF RELATED ART

Modern personal computers and electronic mobile devices typicallyimplement a graphical user interface (GUI) to enable communicatingcommands to the device. Current typical GUI systems enable users toselect objects and initiate actions using a pointer device bypositioning a cursor over a portion of the display and selecting an itemby pressing a button. Such GUI systems allow users to selectapplications, file folders, and files in a menu display (e.g., “desktop”display), and individual words, phrases, graphics, selections of music,selections of graphics, etc. within a document or application. InWindows® type GUI systems, a user can take an action on an object orselection by “highlighting” the item, such as by positioning thepointing device over the item and clicking on mouse button (forexample), and then entering a command on a keyboard. Highlighted objectsor selections can also be “dragged” by moving the pointing device whileholding down the selection button and “dropped” by releasing theselection button to move the object or selection to another location inthe document or display.

In typical GUI systems, users may also implement action on selectedfiles or objects by clicking on a button (e.g., a right mouse button) tobring up a menu of options suitable for the selected object or selection(e.g., copy, cut, paste, open, run, etc.) that can then be selected bypositioning the cursor with the pointing device and clicking a button.In some applications and in some GUI systems (e.g., the Leopard®operating system by Apple Computer, Inc.) a user can initiate an action(e.g., copy, cut, paste, open, run, etc.) by “dragging & dropping” aselected file, application or object onto a suitable icon displayed onthe screen (i.e., an icon associated with a particular operation, suchas a trashcan icon which is associated with a delete function). Forexample, to designate a file for deletion in such a GUI system a usermay select the file with a mouse (i.e., by “clicking on” the file icon),drag it over a trashcan icon, and “drop” the file onto the icon byreleasing the mouse button.

SUMMARY OF THE INVENTION

In one aspect, a method for providing menu icons in a graphical userinterface (GUI) is presented. The method may include detecting a selectand drag event of the object, determining a type of the selected object,identifying actions or functions that are appropriate for the type ofthe selected object and displaying one or more pop-up menu iconsassociated with the identified actions or functions appropriate for theselected object. The method may further include positioning the pop-upmenu icons in a predefined location on a GUI display, positioning thepop-up menu icons in close proximity to the selected object within a GUIdisplay, performing an action or function on the selected object inresponse to the selected object being released when positioned over oneof the pop-up menu icons corresponding to the action or function, orterminating display of the one or more pop-up menu icons in response tothe selected object being dragged beyond an area associated with the oneor more pop-up menu icons. The object may include at least one of anapplication, file, file folder, text selection within a document, imageselection within a document, portion of a an audio file and portion of avideo file. The method may further include registering an applicationwith the GUI including identifying one or more pop-up menu icons whichshould be displayed when an icon associated with the application isselected and dragged within the GUI.

In another aspect, a computer is presented including a processor, adisplay coupled to the processor, and a memory coupled to the processor.The processor of the computer may be configured to detect a select anddrag event of the object, determine a type of the selected object,identify actions or functions that are appropriate for the type of theselected object, and display on the display one or more pop-up menuicons associated with the identified actions or functions appropriatefor the selected object. The processor may also be configured toposition the pop-up menu icons in a predefined location on the GUIdisplay, position the pop-up menu icons in close proximity to theselected object within the GUI display, perform an action or function onthe selected object in response to the selected object being releasedwhen positioned over one of the pop-up menu icons corresponding to theaction or function, terminate display of the one or more pop-up menuicons in response to the selected object being dragged beyond an areaassociated with the one or more pop-up menu icons, or register anapplication with the GUI including identifying one or more pop-up menuicons which should be displayed when an icon associated with theapplication is selected and dragged within the GUI. The object mayinclude at least one of an application, a file, file folder, textselection within a document, image selection within a document, portionof a an audio file and portion of a video file.

In another aspect, a computer is presented including a means fordetecting a select and drag event of the object, a means for determininga type of the selected object, a means for identifying actions orfunctions that are appropriate for the type of the selected object, anda means for displaying on the display one or more pop-up menu iconsassociated with the identified actions or functions appropriate for theselected object. The computer may include a means for positioning thepop-up menu icons in a predefined location on the GUI display, a meansfor positioning the pop-up menu icons in close proximity to the selectedobject within the GUI display, a means for performing an action orfunction on the selected object in response to the selected object beingreleased when positioned over one of the pop-up menu icons correspondingto the action or function, a means for terminating display of the one ormore pop-up menu icons in response to the selected object being draggedbeyond an area associated with the one or more pop-up menu icons, or ameans for registering an application with the GUI including identifyingone or more pop-up menu icons which should be displayed when an iconassociated with the application is selected and dragged within the GUI.The object may include at least one of an application, a file, filefolder, text selection within a document, image selection within adocument, portion of a an audio file and portion of a video file.

In another aspect, a computer program product is presented, in which thecomputer program product has a computer readable medium that may includean instruction to detect a select and drag event of the object, aninstruction to determine a type of the selected object, an instructionto identify actions or functions that are appropriate for the type ofthe selected object and an instruction to display one or more pop-upmenu icons associated with the identified actions or functionsappropriate for the selected object. The computer-readable medium mayfurther include an instruction to position the pop-up menu icons in apredefined location on a GUI display, an instruction to position thepop-up menu icons in close proximity to the selected object within a GUIdisplay, an instruction to perform an action or function on the selectedobject in response to the selected object being released when positionedover one of the pop-up menu icons corresponding to the action orfunction, an instruction to terminate display of the one or more pop-upmenu icons in response to the selected object being dragged beyond anarea associated with the one or more pop-up menu icons, or aninstruction to register an application with the GUI includingidentifying one or more pop-up menu icons which should be displayed whenan icon associated with the application is selected and dragged withinthe GUI. The computer-readable medium may cause a processor of acomputer to perform steps such that the object may include at least oneof an application, file, file folder, electronic document, textselection within a document, image selection within a document, portionof an audio file and portion of a video file.

Additional aspects will be apparent in the foregoing description andequivalents thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and constitutepart of this specification, illustrate exemplary aspects of theinvention. Together with the general description given above and thedetailed description given below, the drawings serve to explain featuresof the invention. In the figures, like reference numerals refer to likeparts throughout the various views unless otherwise indicated.

FIG. 1 is a process flow diagram of a prior art drag & drop GUIinterface function;

FIG. 2 is a process flow diagram of context-dependent pop-up menufunctionality according to an aspect;

FIG. 3 is a process flow diagram of example steps that may beimplemented to generate content-specific pop-up menus;

FIGS. 4-27 are illustrations of graphical user interface displaysshowing example displays of context-dependent pop-up menus in responseto various select and drag operations according to various aspects;

FIGS. 28A-28D are illustrations of an alternative aspect graphical userinterface displays of a context-dependent pop-up menu activated inresponse to various select and drag operations;

FIG. 29 is an example data table suitable for defining context-dependentpop-up menus for various applications, file types and objects accordingto an aspect;

FIG. 30 is a component block diagram of a mobile device suitable forimplementing the various aspects;

FIG. 31 is a component block diagram of a computer suitable forimplementing the various aspects.

DETAILED DESCRIPTION

Various aspects will be described in detail with reference to theaccompanying drawings. References made to particular examples andimplementations are for illustrative purposes and are not intended tolimit the scope of the invention or the claims.

As used herein, the terms “computer,” “computing device” or “mobiledevice” refer to any one or all of personal computers, notebookcomputers, cellular telephones, personal data assistants (PDA's),palm-top computers, wireless electronic mail receivers and cellulartelephone receivers (e.g., the Blackberry® and Treo® devices),multimedia Internet-enabled cellular telephones (e.g., the iPhone®), andsimilar personal electronic devices which include a programmableprocessor, memory and a connected or integral display configured with agraphical user interface (GUI). In a preferred aspect, the electronicdevice is a cellular telephone with a display configured with agraphical user interface.

As used herein, a “touchscreen” is a touch input device including adisplay which can detect the presence and location of a touch within thedisplay and accept such direct onscreen input. As used herein, a“touchpad” is a touch input device including a specialized surface thatcan translate the motion and position of a user's fingers or a stylus toa relative position on screen. A touchpad, for example, can beimplemented on any surface of an electronic device outside the imagedisplay area. Touchscreens and touchpads may be integral parts of anelectronic device, such as a touchscreen display, or a separate modulewhich can be coupled to the electronic device by a wired or wirelessdata link, such as an electronic pen and tablet.

As used herein, a “pointing device” refers to any human-computerinterface device capable of receiving physical inputs from a user thatcan be translated into locations and actions in a GUI. Common pointingdevices include a mouse, touchscreen display, touchpad (as typicallyemployed on some notebook computers), joystick, pointing stick,multi-direction rocker switch (as typically employed on some cellulartelephones), trackball mouse, and electronic pen and tablet. Futurepointing devices that may be developed and would be encompassed in thevarious aspects also include large touch sensitive display panels, eyetrackers, and electronic gloves. Pointing devices are typicallyintegrated with GUI software to position and control a cursor whichappears on the display and moves in conjunction with inputs to thepointing device.

For simplicity of description, the functionality that enables users totake action upon a file or object by selecting it, dragging it acrossthe display and dropping it on menu icon corresponding to an action orfunction is referred to as “drag & drop GUI functionality.” While theusefulness of GUI interfaces is well known, drag & drop GUIfunctionality has also been shown to be very useful in certain devicesand implementation, such as in applications where pointing devices donot have multiple buttons or do not have any buttons at all (e.g.,touchscreen displays). The ability to take an action on a selected itemmerely by dragging it to an icon and releasing it permits users toperform complex tasks without having to touch a keyboard or enteringcommands. In computers and mobile devices which have few buttons andrely upon a touchscreen interface (e.g., table PC's and touchscreenmobile devices), the drag & drop GUI gives users a powerful andintuitive interface for accessing and manipulating files and objectswith one hand.

Nevertheless, the drag & drop GUI functionality suffers from thelimitation that the menu icons must appear on the screen along withother items in the menu display or the document itself. As a result,valuable display “real estate” is taken up by the icons associated withfile and object manipulation actions. This requirement may limit theusefulness of the drag & drop GUI in computing devices with smalldisplays, which may be in mobile devices like cellular telephones. Insuch implementations, the screen is so small that including allfile/object manipulation icons within the display will leave little roomfor files or documents. In computers with larger displays, the drag &drop GUI requires the user to move the item to wherever the appropriateicon is located; in a large display, this may require dragging an itemacross the entire screen.

The various aspects provide a context-dependent pop-up menu icon withina GUI system that is activated (i.e., displayed or caused to “pop-up”within the display) by a ‘select & drag’ operation to display actionmenus or destinations for the selected file or object. When a userselects a particular file or content object (e.g., a word, phrase, orselection of an image) within a document and begins to drag theselection across the display using a pointing device, a set of one ormore menu icons is presented (i.e., pop-up) corresponding to action orfunction options that the user may implement for the selection. Thepresented menu icons may correspond to specific functions or actionsthat are available and can be applied to the particular type of file,object or content that has been selected and dragged. Such pop-up menuicons may be generated for practically anything that can be selected anddragged by a user, who is manipulating a pointing device within a GUI,including for example application icons, file icons, and file foldericons within a GUI display, as well as text selections, numericselections, portions of a spread sheet, graphics, portions of graphics,images, portions of images, audio files, portions of audio files, videofiles, and portions of video files within an electronic document. Forease of reference, a selection by a pointing device, be it anapplication, file, folder or a selection within a document, is referredto herein as an “object.” Thus, the word “object” herein is intended torefer to something selected by manipulation of a pointing device withina GUI system, and should not be interpreted to be limited to programmingobject in an object-oriented programming environment.

The pop-up menu icons may be displayed in fixed positions on the displayor in close proximity to the selection in a consistent manner in termsof position and content. Thus, particular menu icons may consistentlyappear in the same relative position on the display (e.g., a particularcorner of the display) or same relative position with respect to theselection (e.g., to the left or right and above or below the selection).To avoid cluttering the display, menu icons may appear only during aselect & drag operation. Thus, a full set of function icons may beprovided for drag & drop GUI functionality without cluttering thedisplay with icons that are used only infrequently. While menu icons mayappear only during a select & drag operation, the presented icons may beremoved when the user drags the selected file or object beyond the areaof the presented icons. Thus, if a user is merely dragging a word orphrase to a new location in a document, the drag & drop GUI menu iconswill quickly disappear as the drag operation proceeds beyond the icons.In an aspect, new applications can register with the operating system inorder to specify icons to be presented when the files of thecorresponding file type are selected in a menu or when specified contentis selected within the application. In this manner, platform-consistent,context-dependent pop-up menu icons can be provided for all applicationsand content.

The functioning and benefits of the various aspects may be understood incontrast to currently known drag & drop GUI functionality, which isillustrated in the process flow diagram shown in FIG. 1. To implement aprocess on an object using currently known drag & drop GUIfunctionality, icons corresponding to various actions are presented onthe GUI display, block 2. For example, a typical drag & drop capable GUIincludes a trashcan icon associated with the function of designating afile for deletion. The GUI is configured to detect when a particularfile or object is selected by a pointing device, block 4, such as bydetecting when a mouse button is pressed and determining the iconcorresponding to the current cursor location on the display. The GUI maythen track the movement of the selected object with respect to thefunction icons presented on the display, block 6. Movement tracking ofthe selected object, block 6, may continue until the object or file isreleased by the user releasing the mouse button. For example, the GUImay continually decision block whether the mouse button has beenreleased, decision block 8, and continue to track the movement if it hasnot been released. Once the object or file is released (i.e., decisionblock 8=“Yes”), the GUI may compare the location of the cursor at thetime of release to the positions of the various menu icons to determineif the object or file was “dropped” onto one of the icons, decisionblock 10. If the cursor location does not correspond to an iconlocation, then the GUI may simply reposition the selected file or objecticon on the display, block 12. If the cursor location does correspond toan icon location (i.e., decision block 10=“Yes”), the GUI system maydetermine the functionality associated with that particular iconlocation and then implement that functionality on the selected file orobject, block 14. If the functionality associated with the particularicon location is not available to the selected file or object, then noactions are taken. Thus, in previously known drag & drop GUI systems,the available menu icons were presented at all times on the display.

In contrast to the prior art, FIG. 2 is a process flow diagram of anaspect that presents menu icons only during a select & drag operation.This process flow may be implemented as part of a GUI system, such as asubroutine that provides a part of the GUI functionality. In thisaspect, the GUI will detect when a user clicks on a file or contentobject and begins to drag the selection across the display bymanipulating a pointing device, block 20. Detection of file or contentobjects selections and manipulations from pointing device inputs mayemploy any methods well-known in other GUI systems. When a select & dragoperation is detected, the GUI system determines the type of file orcontent object that has been selected, block 22. This block may involveexamining a file type extension of a selected file or examination of theparticular content object selected, such as using methods describedbelow with reference to FIG. 3. Having determined the type of file orobject selected, the GUI system may determine whether there are anyactions or functions available or suitable for the selection, decisionblock 24. If there are no actions or functions that can be implementedon the selection (i.e., decision block 24=“No”), no pop-up menu iconswill be generated and the pop-up icon generation routine may be ended,block 26, with processing continuing according to the rest of the GUIfunctionality. If an action or function can be implemented on theselected file or object (i.e., decision block 24=“Yes”), the GUI systemmay identify the particular actions or functions icons that are relevantto the selection, block 28. As described in more detail below, thisprocess may be implemented in software as a series of conditionaloperations as described with reference to FIG. 3, or as a lookup processin conjunction with a lookup data table as described with reference toFIG. 29. Once the actions or functions relevant to the selection hadbeen identified, the GUI system may generate and present a display ofthose menu icons corresponding to those relevant actions or functions,block 30. In one aspect, the generated menu icons are presented in aparticular location on the display, such as in the four corners and/orother locations about the display periphery. In another aspect, thegenerated menu icons are presented in close proximity to the initiallocation of the selection, with their relative position depending uponthe type of menu icon and the position of the cursor on the display.

With the pop-up menu icons presented on the display, the GUI system maytrack the movement or position of the selection as it is dragged acrossthe display with respect to the icons, block 32. The generated menuicons do not move once they appear on the display, so the movement ofthe selection can be tracked with respect to those icons. As theselection movement is tracked, the GUI system may frequently determinewhether the selection has been dragged beyond the area of the menu icons(particularly in the aspects which present the icons in close proximityto the selection), decision block 34. If the selection has been draggedbeyond the area of the menu icons (i.e. decision block 34=“Yes”), themenu icons may be deleted from the display and the pop-up menu iconroutine terminated, block 26, with processing continuing according tothe rest of the GUI functionality.

If the selection has not been dragged beyond the area of the menu icons(i.e., decision block 34=“No”), the GUI system may determine whether theselection has been released (e.g., the mouse button has been released),decision block 35. If the selection has not been released (i.e.,decision block 35=“No”), the GUI system may continue to track movementof the selection, block 32. Once the selection is released (i.e.,decision block 35=“Yes”), the GUI may compare the location of the cursorat the time of release to the positions of the various menu icons todetermine if the object or file was “dropped” onto one of the pop-upmenu icons, decision block 36. If the cursor location does notcorrespond to an icon location (i.e., decision block 36=“No”), the menuicons may be deleted from the display and the pop-up menu icon routineterminated, block 26, with processing continuing according to the restof the GUI functionality, such as repositioning the selected file orobject icon on the display. If the cursor location does correspond to aparticular pop-up menu icon location (i.e., decision block 36=“Yes”),the GUI system may perform the action or function corresponding to thatmenu icon (if there is no application function), or notify anapplication of the action or function corresponding to that menu icon sothat the appropriate action or function can be implemented on theselection, block 38. Once the action or function corresponding to theparticular menu icon is implemented, the menu icons may be deleted fromthe display and the pop-up menu icon routine terminated, block 26, withprocessing continuing according to the rest of the GUI functionality.

A variety of methods may be implemented to determine the type of file orobject selected and which pop-up menu icons should be presented in thedisplay. FIG. 3 illustrates example process blocks that may beimplemented to accomplish this according to an aspect. In the aspectillustrated in FIG. 3, determination of the selection and identificationof suitable pop-up menu icons is accomplished using conditional logic.This aspect is presented by way of example, and is not intended to limitthe scope of the invention because a number of other processing routinesmay be implemented to accomplish the functionality associated withblocks 22 through 30 described above with reference to FIG. 2.

Referring to FIG. 3, the GUI system may determine whether the selectionwas made within a menu context (e.g., within a desktop or file list) orwithin an application or document, decision block 40. If the selectionwas made within a document, as may be the case when a word or phrase hasbeen selected within a word processing application, the GUI system maydetermine whether the selection includes text, decision block 42. If theselection includes text (i.e., decision block 42=“Yes”), the selectionmay be parsed to determine the type or content of the text, block 44.For example, the selection may be parsed and compared to templates torecognize names, telephone numbers, e-mail addresses, Internet addresses(e.g., a URL), mailing addresses, currency or other numeric values, etc.If the selection does not match a predefined type, then it may beassumed to be ordinary text. Any of a variety of known methods forparsing and recognizing text content may be implemented in this block44.

After determining the type of text contained within the selection, theGUI system may determine the actions or functions that are appropriatefor or available to act on the determined text type, block 46. Forexample, if the selected text includes a name, the name may be copied,cut, added to a contacts database or used to look up a contact in thecontacts database, for example. Also, other functions may be availableand appropriate for a name, such as conducting an Internet search (e.g.,conducting a Google search on the name). The available actions orfunctions may be determined using conditional logic, such as in a seriesof “if [type], then [actions list]” software instructions.Alternatively, a table-look up algorithm may be implemented as describedbelow with reference to FIG. 29. Once the available and appropriateactions or functions are identified for the selected text, the GUI maygenerate a display of the corresponding pop-up menu icons, block 48.Once the display is generated, the processing may proceed to trackmovements of the selection as described above with respect to block 32in FIG. 2.

For purposes of this example, if the selection is not text (i.e.,decision block 42=“No”), there are no actions or functions available toact on the selection, so no menu icons may be displayed and the pop-upmenu icon routine terminated, FIG. 2 block 26, with processingcontinuing according to the rest of the GUI functionality. In a typicalimplementation, functions or actions may be available for imageselections (e.g., selections from PDF, TIF or JPG images), such as copy,cut, save, etc., so additional determinations may be applied to thedocument selection to identify such document types and generateappropriate pop-up menu icons.

Returning to decision block 40, if the item was selected in a menu ordesktop setting, the item will be a file, a folder or an application. Inthat case, the selection may be further examined by the GUI system todetermine if it is a file folder, decision block 50. If the selection isa file folder (i.e., decision block 50=“Yes”), then appropriate actionsor functions include open, copy, cut (or delete) and send (e.g., as anattachment to an e-mail). Accordingly, the GUI system may generate adisplay of pop-up menu icons for open, copy, cut (or delete) and send,block 52. If the selection is not a file folder (i.e., decision block50=“No”), the GUI system may determine if the selection is a file (e.g.,a text, image, audio, video, PDF, or other type that can opened or actedupon by an application), decision block 54.

If the selection is a file, the GUI system may further examine theselection to determine the file type, block 56. For example, the GUIsystem may examine a file extension to determine the file type (e.g., bymatching the extension to a table of extensions). Using the determinedfile type, the GUI system can then determine the actions or functionsappropriate and available for the file type. For example, if the file isa text file, it may be copied, cut (or deleted), sent, or opened by atext editor. Also, other functions may be available for a text file,such as spell checking. As another example, if the file is an MP3 file,it may be copied, cut (or deleted), sent or played. Having determinedthe actions or functions available and appropriate for the file type,the GUI system can generate a display of pop-up menu icons correspondingto those actions or functions, block 60.

If the selection is not a file (i.e., decision block 54=“No”), the GUIsystem may examine the selection further to determine if it is anexecutable application, decision block 62. If selection is an executableapplication (i.e., decision block 62=“Yes”), the available actions orfunctions may be limited to open, in which case the GUI system cangenerate a display of a pop-up menu icon corresponding to the openactions, block 64. If the selection is not an executable file (i.e.,decision block 62=“No”), no action or function may be available, so nomenu icons may be displayed and the pop-up menu icon routine terminated,FIG. 2 block 26, with processing continuing according to the rest of theGUI functionality. If pop-up menu icons are displayed (as may occur ifany one of decision blocks 50, 54, 62=“Yes”), processing may proceed totrack movements of the selection as described above with respect toblock 32 in FIG. 2.

The functioning and functionality of the various methods may be furtherunderstood by reference to examples illustrated in FIGS. 4-27. Each ofthese figures illustrate a display 70 of a computer or mobile device,with the displays illustrating various blocks in the implementation ofvarious aspects as might be viewed by a user. In these examples thelarge arrow 71 represents the cursor of the GUI interface.

In the example illustrated in FIG. 4, the display 70 shows a portion ofa text document in which the user has used a pointing device to select aword 72, namely “jumped.” FIG. 4 illustrates how the display 70 willappear before pop-up menu icons are displayed, showing how the display70 is not cluttered with menu icons.

FIG. 5 illustrates the example shown in FIG. 4 after the user hasdragged the selected word 72 which triggers pop-up menu icons 74, 76, 78and 80 to be displayed. In this example the word 72 “jumped” is a simpletext item, so the actions or functions available for text are, in thisexample, copy 74, cut 76, spell check 78 and look up in Google® 80.Also, this example shows the aspect in which the pop-up menu icons arepresented in standard locations in the display 70, namely the fourcorners of the display 70 in this example.

FIG. 6 illustrates how the display 70 will occur if the user drags theselected word 72 and releases (“drops”) it on the spell checkingfunction icon 78. This action causes the other pop up menu icons 74, 76,80 to disappear as shown, and the spell checking function to beperformed on the selected word.

FIG. 7 illustrates the aspect in which the pop-up menu icons arepresented in close proximity to the selection after the user has draggedthe selected word 72 to trigger display 70 of the pop-up menu icons 74,76, 78 and 80. As in FIG. 5, in this example the word 72 “jumped” is asimple text item, so the actions or functions available for text arecopy 74, cut 76, spell check 78 and look up in Google® 80. As FIG. 7illustrates, the pop-up menu icons appear close by the selection so auser can select an action or function by dragging the selection a shortdistance. While the pop-up menu icons 74, 76, 78 and 80 appear over thedocument on the display 70, the icons will appear only until a selectionis made, and therefore do not block the display 70 except when the menuicons are relevant.

FIG. 8 illustrates how the display 70 will occur if the user drags theselected word 72 and releases (“drops”) it on the spell checkingfunction icon 78. This action causes the other pop-up menu icons 74, 76,80 to disappear as shown, and the spell checking function to beperformed on the selected word.

FIG. 9 illustrates how the display 70 will occur if the user drags theselected word 72 beyond the area of the pop-up menu icons shown in FIG.7. This action by the user causes the pop-up menu icons to be deletedand the GUI system to be returned to normal functioning (see decisionblock 34 and block 26 in FIG. 2). As FIG. 9 shows, this functionalityallows a user to cause the pop-up menu icons to go away simply by movingthe selection a certain distance, after which the selection canrepositioned and dropped without inadvertently activating an undesiredmenu. Thus, the pop-up menu icons do not prevent users from utilizingdrag & drop processes available in conventional GUI systems.

FIG. 10 illustrates an example of the display 70 in which a user hasselected a name 82 but has not yet dragged the selection. FIG. 11illustrates the display 70 after the user has dragged the name 82 asmall distance triggering display of the pop-up menu icons 74, 76, 80,84, 86, 88. In this example, the GUI system recognizes that the selectedtext is a name and accordingly displays pop-up icons that are suitablefor a name, including copy 74, cut 76, add to a contacts database 84,look up in the contacts database 86, call 88 and, in this example, lookup in Google® 80. FIG. 11 also illustrates how the copy 74, cut 76 andGoogle® 80 menu icons are positioned in approximately the same positionrelative to the selection as shown in the example in FIG. 7. Thus, sincecopy 74 and cut 76 are available for both text and name objects, thesemenu icons appear in the same relative position. Thus, a user will learnthat these functions can be accomplished on a selection by moving theselection in the same directions even though the selections includedifferent types of content. FIG. 11 also illustrates how the placementof the pop-up menu icons may be adjusted to fit within the display 70when the selection appears near a display boundary.

FIG. 12 illustrates an example of the display 70 in which a user hasselected a telephone number 90 but has not yet dragged the selection.FIG. 13 illustrates the display 70 after the user has dragged thetelephone number 90 a small distance triggering display of the pop-upmenu icons 74, 76, 84, 86, 88. In this example, the GUI systemrecognizes that the selected text is a telephone number and accordinglydisplays pop-up icons that are suitable for a telephone number,including copy 74, cut 76, add to a contacts database 84, look up in thecontacts database 86, and call 88.

As a further example, FIG. 14 illustrates an example of the display 70in which a user has selected an e-mail address 92 but has not yetdragged the selection. FIG. 15 illustrates the display 70 after the userhas dragged the e-mail address 92 a small distance triggering display ofthe pop-up menu icons 74, 76, 84, 86, 94. In this example, the GUIsystem recognizes that the selected text is an e-mail address andaccordingly displays pop-up icons that are suitable for an e-mailaddress, including copy 74, cut 76, add to a contacts database 84, lookup in the contacts database 86, and send an e-mail 94.

As a further example, FIG. 16 illustrates an example of the display 70in which a user has selected postal address 96 but has not yet draggedthe selection. FIG. 17 illustrates the display 70 after the user hasdragged the postal address 96 a small distance triggering display of thepop-up menu icons 74, 76, 84, 86, 98. In this example, the GUI systemrecognizes that the selected text is a postal address and accordinglydisplays pop-up icons that are suitable for a postal address, includingcopy 74, cut 76, add to a contacts database 84, look up in the contactsdatabase 86, and look up the address in a mapping program, such asGoogle Earth™ 98.

FIG. 18 illustrates an example of the display 70 when a file 100 isselected within a menu setting, such as a desktop. In this example, thefile 100 is an MP3 file which has been selected, but not yet dragged.

FIG. 19 illustrates how the display 70 will include pop-up menu icons74, 76, 94, 102, 104 when the MP3 file 100 has been dragged a shortdistance. In this example, the GUI system has determined that the file100 is an MP3, and accordingly has generated pop-up menu icons foractions or functions appropriate for a music file, namely copy 74, cut76, send 94 (as in an e-mail attachment), open 102 and play 104. In asituation where the computer or mobile device includes multiple musicplayers, separate player activation menu icons may be presented so thata user can select a particular player simply be dragging the selectedfile 100 to the desired player icon.

FIG. 20 illustrates how the display 70 will appear when the user dragsthe selected file 100 and “dropped” it on the player icon 104. Thisaction causes the other pop-up menu icons to be removed while the mediaplayer application launches.

FIG. 21 illustrates how the display 70 will appear when the user dragsthe selected file 100 beyond the area of the pop-up menu icons. Thisaction causes the pop-up menu icons to be removed and GUI functioning toproceed without the pop-up icons.

FIG. 22 illustrates an example of the display 70 when a word processingapplication 106 is selected on the desktop before it is dragged. FIG. 23illustrates how the display 70 will include an open pop-up menu icon 102when the word processing application 106 has been dragged a shortdistance. For purposes of example, FIG. 23 illustrates an implementationin which a word processing application only has a single availablefunction, namely open. In some implementations, an application may haveother available functions, such as cut (or delete).

FIG. 24 illustrates an example of the display 70 when a file folder 108is selected on the desktop before it is dragged. FIG. 26 illustrates howthe display 70 will include pop-up menu icons 74, 76, 94, 102 when thefile folder 108 has been dragged a short distance. In this example,available and appropriate actions or functions for a file folder includecopy 74, cut (or delete) 76, send 94 (as in an e-mail attachment) andopen 102.

FIG. 26 illustrates an example of the display 70 when a word processingfile 110 is selected on the desktop before it is dragged. FIG. 27illustrates how the display 70 will include pop-up menu icons 74, 76,78, 94, 102 when the word processing file 110 has been dragged a shortdistance. In this example, available and appropriate actions orfunctions for a word processing file 110 include copy 74, cut 76 (ordelete), spell checking 78, send 94 (as in an e-mail attachment) andopen 102.

In alternative aspect, the pop-up menu icons may be configured to appearin a peripheral menu panel 112 surrounding the periphery of the display70, an example of which is illustrated in FIGS. 28A and 28B. Referringto FIG. 28A, a peripheral menu panel 112 may be generated in response toan object, such as text 72, is selected and dragged. Menu icons 74, 76,78, 80 available and appropriate for the selected object may bedisplayed in the peripheral menu panel 112. Such a peripheral menu panel112 may appear over a portion of the display 70. In an aspect, the menuicons appear in the same portions of the panel so that icons availableand appropriate for a selected object are displayed within their panelportions while unavailable icons result in blank portions. This isillustrated FIG. 28B in which the ‘Add to contacts’ menu icon 84 and“Look up” menu icon 86 are appropriate for a selected name 82. Thesepop-up menu icons 84, 86 appear in portions of the pop-up peripheralmenu panel 112 that are blank in the selected text object example shownin FIG. 28A.

In alternative aspect, the pop-up menu icons may be configured to appearin an arc or circle surrounding a selected and dragged object or file,an example of which is illustrated in FIGS. 28C and 28D. Referring toFIG. 28C, a circular menu wheel 114 may be generated in response to anobject, such as text 72, is selected and dragged. Menu icons 74, 76, 78,80 available and appropriate for the selected object may be displayed inthe circular menu wheel 114. Such a pop-up circular menu wheel 114 mayappear over a portion of the display 70. In an aspect, the menu iconsappear in the same sector so that icons available and appropriate for aselected object are displayed within their sectors while unavailableicons result in blank sectors. This is illustrated FIG. 28D in which the‘Add to contacts’ menu icon 84 and “Look up” menu icon 86 areappropriate for a selected name 82. These pop-up menu icons 84, 86appear in sectors of the pop-up circular menu wheel 114 that are blankin the selected text object example shown in FIG. 28C.

As the foregoing examples illustrate, the various aspects provide aflexible menu icon system that presents icons only when needed and onlyas available for a particular selection.

In an aspect, a data table may be stored in memory and used by the GUIsystem to determine which (if any) pop-up menu icons should be displayedfor each type of selection (i.e., folder, file, application, or objecttype). FIG. 29 illustrates a portion of an example data table 120 thatmay be used in conjunction with the various aspects. Such a data table120 may include a plurality of records (rows) that each includes anumber of data records (columns 121-129). In this example, a data recordis provided for each type of file, item or object that may be selectedand dragged (see column 121), along with flags that may be set toindicate whether particular menu icons should be displayed in the eventof a select & drag operation on a particular object. By storing a seriesof flags in a data table in this manner a large number of different ofmenu icons supporting a large number of different objects andapplications can be supported in a compact and flexible manner.

In an aspect, when the GUI system determines the type of the selectedfile or object (block 22 in FIG. 2), this determination can be used as alook up value to determine the appropriate record to access in the datatable 120, and then use the flags in that record to determine whichpop-up menu icons to display. When the menu icons are identified, theGUI system can access icon graphic files stored in a different memorylocation to generate the display 70. In alternative aspect, instead ofincluding binary flags each data record 122-129 may include a memorypointer to a memory location containing the graphics for an icon if amenu icon is appropriate.

For example, consider the case of a text item selection (as isillustrated in FIGS. 4-8). The data table 120 includes a “1” or positiveflag in data fields associated with copy 122 and cut 123 for the textdata record, and “0” or negative flags in the other data fields.Similarly, the example data table 120 includes a “1” or positive flag indata fields associated with copy 122, cut 123, contact add 125, contactlook up 126 for the name data record, and “0” or negative flags in theother data fields. Example flag settings for other types of objects andfiles are illustrated in FIG. 29.

The use of a data table 120 to keep track of menu icons also enables anaspect in which applications can register with the operating system toenable context-specific pop-up menu icons. Registration of a newapplication for context-specific pop-up menu icons can be accomplishedby appending to the data table 120 records for the application, as wellas any unique application document object types, which include datafields 122-129 with flags (or memory pointers) indicating which pop-upmenu icons should be displayed when the application and any uniqueobject types are selected and dragged. This is illustrated in FIG. 29 inthe bottom two rows which show for Application X that the open and playmenu icons should be displayed and for Application X unique data objecttype 1 the copy, cut, contact add and send function menu icons should bedisplayed. As indicated by the ellipsis 130 in the columns and rows, anynumber of additional application records can be added and any number ofdata fields (i.e., different menu icons) can be added.

The hardware used to implement the forgoing aspects may be processingelements and memory elements configured to execute a set ofinstructions, wherein the set of instructions are for performing methodblocks corresponding to the above methods. Alternatively, some functionsor methods may be performed by circuitry that is specific to a givenfunction.

The aspects described above may be implemented on any of a variety ofmobile devices. Typically, such mobile devices will have in common thecomponents illustrated in FIG. 30. For example, the portable computingdevices 131 may include a processor 132 coupled to internal memory 133and a display 139. Additionally, the portable computing device 131 willhave an antenna 134 for sending and receiving electromagnetic radiationthat is connected to a wireless data link and/or cellular telephonetransceiver 135 coupled to the processor 132. In some implementations,the transceiver 135 and portions of the processor 132 and memory 133used for cellular telephone communications is referred to as the airinterface since it provides a data interface via a wireless data link.Portable computing devices 139 also typically include a key pad 136 orminiature keyboard and menu selection buttons or rocker switches 137which serve as pointing devices for receiving user inputs forpositioning a cursor within the display 139. Portable computing devices139 may also include a touchpad or touchscreen which may serve as apointing device for receiving user inputs for positioning a cursorwithin the display 139. The processor 132 may further be connected to awired network interface 138, such as a universal serial bus (USB) orFireWire® connector socket, for connecting the processor 132 to anexternal pointing device (e.g., a mouse) or computing device such as apersonal computer 160 or external local area network.

The processor 132 may be any programmable microprocessor, microcomputeror multiple processor chip or chips that can be configured by softwareinstructions (applications) to perform a variety of functions, includingthe functions of the various aspects described above. In some portablecomputing device 131, multiple processors 132 may be provided, such asone processor dedicated to wireless communication functions and oneprocessor dedicated to running other applications. The processor mayalso be included as part of a communication chipset. Typically, softwareapplications may be stored in the internal memory 133 before they areaccessed and loaded into the processor 132. In some mobile devices 131,the processor 132 may include internal memory sufficient to store theapplication software instructions. For the purposes of this description,the term memory refers to all memory accessible by the processor 132,including internal memory 133 and memory within the processor 132itself. Application data files are typically stored in the memory 133.In many mobile devices 131, the memory 133 may be a volatile ornonvolatile memory, such as flash memory, or a mixture of both.

The aspects described above may also be implemented on any of a varietyof computing devices, such as a personal computer 160 illustrated inFIG. 31. Such a personal computer 160 typically includes a processor 161coupled to volatile memory 162 and a large capacity nonvolatile memory,such as a disk drive 163. The computer 13 may also include a floppy discdrive 164 and a compact disc (CD) drive 165 coupled to the processor161. Typically the computer device 13 will also include a pointingdevice such as a mouse 167, a user input device such as a keyboard 168and a display 166. The computer device 13 may also include a number ofconnector ports coupled to the processor 161 for establishing dataconnections or receiving external memory devices, such as a USB orFireWire® connector sockets or other network connection circuits 166 forcoupling the processor 161 to a network. In a notebook configuration,the computer housing includes the pointing device 167, keyboard 168 andthe display 169 as is well known in the computer arts.

The various aspects may be implemented by a computer processor 161executing software instructions configured to implement one or more ofthe described methods. Such software instructions may be stored inmemory 162, 163 as separate applications, or as compiled softwareimplementing an aspect method. Reference database may be stored withininternal memory 162, in hard disc memory 164, on tangible storage mediumor on servers accessible via a network (not shown). Further, thesoftware instructions and databases may be stored on any form oftangible processor-readable memory, including: a random access memory162, hard disc memory 163, a floppy disk (readable in a floppy discdrive 164), a compact disc (readable in a CD drive 165), electricallyerasable/programmable read only memory (EEPROM), read only memory (suchas FLASH memory), and/or a memory module (not shown) plugged into thecomputer 160, such as an external memory chip or a USB-connectableexternal memory (e.g., a “flash drive”) plugged into a USB network port166.

Those of skill in the art would appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the aspects disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. Skilled artisans may implement the describedfunctionality in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the present invention.

The order in which the blocks of a method described above and shown inthe figures is for example purposes only as the order of some blocks maybe changed from that described herein without departing from the spiritand scope of the present invention and the claims.

The blocks of a method or algorithm described in connection with theaspects disclosed herein may be embodied directly in hardware, in asoftware module executed by a processor, or in a combination of the two.A software module may reside in processor readable memory which may beany of RAM memory, flash memory, ROM memory, EPROM memory, EEPROMmemory, registers, hard disk, a removable disk, a CD-ROM, or any otherform of storage medium known in the art. An exemplary storage medium iscoupled to a processor such that the processor can read informationfrom, and write information to, the storage medium. In the alternative,the storage medium may be integral to the processor. The processor andthe storage medium may reside in an ASIC. The ASIC may reside in a userterminal or mobile device. In the alternative, the processor and thestorage medium may reside as discrete components in a user terminal ormobile device. Additionally, in some aspects, the blocks and/or actionsof a method or algorithm may reside as one or any combination or set ofcodes and/or instructions on a machine readable medium and/or computerreadable medium, which may be incorporated into a computer programproduct.

The foregoing description of the various aspects is provided to enableany person skilled in the art to make or use the present invention.Various modifications to these aspects will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other aspects without departing from the spirit or scope ofthe invention. Thus, the present invention is not intended to be limitedto the aspects shown herein, and instead the claims should be accordedthe widest scope consistent with the principles and novel featuresdisclosed herein.

1. A method for providing menu icons in a graphical user interface(GUI), comprising: detecting a select and drag event of the object;determining a type of the selected object; identifying actions orfunctions that are appropriate for the type of the selected object; anddisplaying one or more pop-up menu icons associated with the identifiedactions or functions appropriate for the selected object.
 2. The methodof claim 1, further comprising positioning the pop-up menu icons in apredefined location on a GUI display.
 3. The method of claim 1, furthercomprising positioning the pop-up menu icons in close proximity to theselected object within a GUI display.
 4. The method of claim 1, furthercomprising performing an action or function on the selected object inresponse to the selected object being released when positioned over oneof the pop-up menu icons corresponding to the action or function.
 5. Themethod of claim 3, further comprising terminating display of the one ormore pop-up menu icons in response to the selected object being draggedbeyond an area associated with the one or more pop-up menu icons.
 6. Themethod of claim 1, wherein the object comprises at least one of anapplication, file, file folder, text selection within a document, imageselection within a document, portion of a an audio file and portion of avideo file.
 7. The method of claim 1, further comprising registering anapplication with the GUI including identifying one or more pop-up menuicons which should be displayed when an icon associated with theapplication is selected and dragged within the GUI.
 8. A computer,comprising: a processor; a display coupled to the processor; and amemory coupled to the processor, wherein the processor is configured to:detect a select and drag event of the object; determine a type of theselected object; identify actions or functions that are appropriate forthe type of the selected object; and present on the display one or morepop-up menu icons associated with the identified actions or functionsappropriate for the selected object.
 9. The computer of claim 8, whereinthe processor is configured to position the pop-up menu icons in apredefined location on the GUI display.
 10. The computer of claim 8,wherein the processor is configured with software instructions toposition the pop-up menu icons in close proximity to the selected objectwithin the GUI display.
 11. The computer of claim 8, wherein theprocessor is configured to perform an action or function on the selectedobject in response to the selected object being released when positionedover one of the pop-up menu icons corresponding to the action orfunction.
 12. The computer of claim 10, wherein the processor isconfigured to terminate display of the one or more pop-up menu icons inresponse to the selected object being dragged beyond an area associatedwith the one or more pop-up menu icons.
 13. The computer of claim 8,wherein the processor is configured such that the object comprises atleast one of an application, a file, file folder, text selection withina document, image selection within a document, portion of a an audiofile and portion of a video file.
 14. The computer of claim 8, whereinthe processor is configured to register an application with the GUIincluding identifying one or more pop-up menu icons which should bedisplayed when an icon associated with the application is selected anddragged within the GUI.
 15. A computer, comprising: means for detectinga select and drag event of an object within a graphical user interface(GUI); means for determining a type of the selected object; means foridentifying actions or functions that are appropriate for the type ofthe selected object; and means for displaying one or more pop-up menuicons associated with the identified actions or functions appropriatefor the selected object.
 16. The computer of claim 15, furthercomprising means for positioning the pop-up menu icons in a predefinedlocation on a GUI display.
 17. The computer of claim 15, furthercomprising means for positioning the pop-up menu icons in closeproximity to the selected object within a GUI display.
 18. The computerof claim 15, further comprising means for performing an action orfunction on the selected object in response to the selected object beingreleased when positioned over one of the pop-up menu icons correspondingto the action or function.
 19. The computer of claim 17, wherein theprocessor is configured with software instructions to perform furthersteps comprising terminating display of the one or more pop-up menuicons in response to the selected object being dragged beyond an areaassociated with the one or more pop-up menu icons.
 20. The computer ofclaim 15, wherein the object may comprise of at least one of anapplication, file, file folder, text selection within a document, imageselection within a document, portion of an audio file and portion of avideo file.
 21. The computer of claim 15, further comprising means forregistering an application with the GUI including identifying one ormore pop-up menu icons which should be displayed when an icon associatedwith the application is selected and dragged within the GUI.
 22. Acomputer program product, comprising: a computer-readable medium,comprising: at least one instruction for detecting a select and dragevent of the object; at least one instruction for determining a type ofthe selected object; at least one instruction for identifying actions orfunctions that are appropriate for the type of the selected object; andat least one instruction for displaying one or more pop-up menu iconsassociated with the identified actions or functions appropriate for theselected object.
 23. The computer program product of claim 22, whereinthe computer-readable medium further comprises: at least one instructionfor positioning the pop-up menu icons in a predefined location on a GUIdisplay.
 24. The computer program product of claim 22, wherein thecomputer-readable medium further comprises: at least one instruction forpositioning the pop-up menu icons in close proximity to the selectedobject within a GUI display.
 25. The computer program product of claim22, wherein the computer-readable medium further comprises: at least oneinstruction for performing an action or function on the selected objectin response to the selected object being released when positioned overone of the pop-up menu icons corresponding to the action or function.26. The computer program product of claim 24, wherein thecomputer-readable medium further comprises: at least one instruction forterminating display of the one or more pop-up menu icons in response tothe selected object being dragged beyond an area associated with the oneor more pop-up menu icons.
 27. The computer program product of claim 22,wherein the object may comprise of at least one of an application, file,file folder, electronic document, text selection within a document,image selection within a document, portion of an audio file and portionof a video file.
 28. The computer program product of claim 22, whereinthe computer-readable medium further comprises: at least one instructionfor registering an application with the GUI including identifying one ormore pop-up menu icons which should be displayed when an icon associatedwith the application is selected and dragged within the GUI.